0.2.0: switch to nested layout #40
Conversation
This defines the dimension separator (sometimes called "key separator") for chunks in the NGFF file to be "/" rather than ".". The benefit of storing hierarchically is that for very large volumes the number of files in a directory is limited to `shape / chunk` for a given dimension. close: ome#29
|
The W3 services diff looks good and makes sense to me thanks @joshmoore. From the upstream Zarr perspective, this matches the work that has been engaged to use On the implementation side, various efforts have started and will keep happening to prepare tools to update data writing and reading to handle the new separator- see glencoesoftware/bioformats2raw#80 for instance. My main question for other implementations was the following: what is the most direct way to detect that an image has been stored according to version 0.2 (or later) of this specification and that |
|
Practically, I think an initialization of the zarr store will happen to get access to the metadata and then if it's a version that requires special action (e.g. version == 0.1, therefore hard-code I thought about migrating @constantinpape's changes to this PR, but I wasn't completely sure on their status (i.e. if more are coming). Happy to add those on top of the 0.2/index.bs at any point in the future. |
This also updates the version added to the metadata based on the nested flag. In the future, this logic will need to be more sophisticated. see: ome/ngff#40
I think these changes are done, except for a decision about MUST vs SHOULD for the "axes" field. I am happy to rebase #39 onto the changes here, or change #39 so that we merge it into this PR instead of master. |
|
Going to start merging all the related nested-chunk-storage PRs. In implementing zarr-developers/zarr-python#716, I think I've convinced myself that we will be able to drop this requirement soon, i.e. this specification should be agnostic to the underlying storage, but I'd propose getting this in. Future versions can describe the nested layout as a |
|
@constantinpape : happy to take your wording update on top of 0.2/* as you modify 0.3/* |
0.2.0: switch to nested layout SHA: af5399f Reason: push, by @joshmoore Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sorry, I was busy with other stuff last week... |
This defines the dimension separator (sometimes called
"key separator") for chunks in the NGFF file to be "/"
rather than ".". The benefit of storing hierarchically
is that for very large volumes the number of files in
a directory is limited to
shape / chunkfor a givendimension.
close: #29
Review:
(Note: the review URLs use my branch name and therefore will be updated when I push)
Primary change: